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ABSTRACT 



With a continuing emphasis on improving student reading 
achievement across the nation, many elementary educators are exploring the 
idea of initiating a volunteer reading tutoring program. Although individuals 
associated with tutoring programs may think about implementation issues, 
often the focus is on program location, the recruitment and training of 
tutors, and the selection and scheduling of students- -evaluation of a 
tutoring program is often forgotten. This guide focuses on tutoring through 
the lens of the national America Reads Challenge initiative, and it takes 
program implementation to a deeper level by examining volunteer 
transportation, tutor work ethic, and school expectations. The guide 
summarizes and synthesizes findings from evaluations and provides 
introductory information to funders, program coordinators and staff, school 
administrators, elementary teachers, and tutors about implementing and 
evaluating a volunteer reading tutoring program. It also provides evaluation 
results and accompanying conclusions from five America Reads tutoring 
programs as well as program evaluation recommendations. (Contains 30 notes 
and 13 tables.) (NKA) 
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Executive Summary 



With a continuing emphasis on improving student reading achievement across the nation, many 
elementary educators are exploring the idea of initiating a volunteer reading tutoring program. 
Although individuals associated with tutoring programs may think about implementation issues, 
often the focus is on program location, the recruitment and training of tutors, and the selection 
and scheduling of students. Evaluation of a tutoring program is often forgotten. 

This guide focuses on tutoring through the lens of the national America Reads Challenge 
initiative, and it takes program implementation to a deeper level by examining volunteer 
transportation, tutor work ethic, and school expectations. Also provided are evaluation results 
and accompanying conclusions from five America Reads tutoring programs and program 
evaluation recommendations. 
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Introduction 



The America Reads Challenge, a national campaign initiated in 1997, called for a grassroots 
“citizen army” of parents, teachers, students, senior citizens, and other volunteers to become 
reading tutors to primary students. Federal monies were also allocated to pay work- study 
students at designated colleges and universities to serve as tutors. 

To ensure the quality of volunteer literacy programs, the U.S. Department of Education set aside 
several million dollars for tutor training and support. Through the Regional Educational 
Laboratories (RELs), schools and community agencies were invited to form partnerships and 
apply for funding to start or expand the training component of volunteer reading programs. The 
proposal guidelines did not require projects to evaluate the impact of tutoring services on student 
achievement; however, many programs chose to do so. 

In September 1998, the following nine southeastern programs each received $50,000 in grant 
money to be used from October 1998 to March 1999 to build their reading tutoring programs: 

• Birmingham READS in Birmingham, Alabama 

• Hillsborough Reads in Tampa, Florida 

• Miami Reads Tutorial Project in Miami-Dade County, Florida 

• Southwest Georgia Regional Training Project based in Albany, Georgia 

• City Wide Readers in Atlanta, Georgia 

• Mississippi Reads in Jackson, Mississippi 

• School Reading Partners in Chapel Hill-Carrboro, North Carolina 

• America Reads through Family Literacy in Gastonia, North Carolina 

• Reading Soul Mates in Charleston, South Carolina 

Of the nine grantees, five went beyond the requirements of the SERVE grant, by collecting 
program evaluation data during the 1998-99 school year and making the results available to 
SERVE. 1 No evaluation guidelines were provided, so the designs varied. 

The purpose of this guide is to summarize and synthesize findings from these evaluations 
and to provide introductory information to funders, tutoring program coordinators and 
staff, school administrators, elementary schoolteachers, and tutors about implementing and 
evaluating a volunteer reading tutoring program. 

The most recent results of the National Assessment of Educational Progress (NAEP) confirmed 
that public school children in the southeastern United States are in need of assistance to develop 
and strengthen basic literacy skills. According to the NAEP 1998 Reading Report Card for the 



1 The five projects that provided evaluation data for this report are Hillsborough Reads, the Miami Reads Tutorial 
Project, Mississippi Reads, Reading Soul Mates, and School Reading Partners. Contact information for each of these 
projects appears at the end of this report. 



Nation and the States, 2 in five of the six states in the SERVE region, both fourth- and eighth- 
graders performed below the national average on the 1998 NAEP reading assessment. 3 

As Table 1 shows, in the six SERVE states, 38-52% of sampled fourth-graders in the region 
scored “below basic.” Another 30-34% of students were assessed as reading only at the “basic” 
or partial mastery level. Only 18 to 28% of students in the region scored at the “proficient” level. 



Table 1 

Percentage of Fourth-Graders within Each NAEP Achievement Level 



SERVE States 


Below Basic 


Basic 


Proficient 


Advanced 


Alabama 


44 


32 


19 


5 


Florida 


46 


31 


18 


5 


Georgia 


45 


31 


19 


5 


Mississippi 


52 


30 


15 


3 


North Carolina 


38 


34 


22 


6 


South Carolina 


45 


33 


18 


4 


National Average 


39 


31 


23 


6 



Table 2 tells a similar story about eighth-grade reading levels in the SERVE states, with 38-52% 
of students scoring below the basic level. 



Table 2 



Percentage of Eighth-Graders within Each NAEP Achievement Level 



SERVE States 


Below Basic 


Basic 


Proficient 


Advanced 


Alabama 


34 


45 


20 


1 


Florida 


35 


42 


22 


1 


Georgia 


32 


43 


24 


1 


Mississippi 


39 


42 


18 


1 


North Carolina 


24 


45 


29 


2 


South Carolina 


35 


43 


21 


1 


National Average 


28 


41 


28 


2 



State-by-state data are not available for the twelfth-grade reading assessment, although data have 
been compiled for four broad regions of the country: Central, Northeast, Southeast, and West. 
The SERVE states are a subset of the Southeast as defined for the NAEP. 4 In 1998, twelfth-grade 
students in the Southeast scored lower than their counterparts in the other three NAEP regions. 

The America Reads projects identified in this report all provide tutoring services to primary- 
grade school children. The preceding data demonstrate that students in the SERVE region fall 
behind the rest of the nation at an early age. The America Reads projects at SERVE are designed 
to provide support to struggling young readers in hopes of reversing this trend. 



2 Donahue, Patricia L., Voelkl, Kristin E., Campbell, Jay R., and Mazzeo, John (March 1999). NAEP 1998 Report 
Card for the Nation and States. Washington, DC: National Center for Education Statistics, U.S. Department of 
Education. Available at http://nces.ed.gov/nationsreportcard/Dubs/mainl998/1999500.pdf . 

3 Of the six states in the SERVE region, only North Carolina performed “at or around the national average.” 

4 In addition to the six states in the SERVE region, NAEP results for the Southeast include performance data for 
Arkansas, Kentucky, Louisiana, Tennessee, Virginia, and West Virginia. 
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Description and Evaluation Overview of Tutoring Programs 



The five projects that are the subject of this report all share the goal of improving primary 
students’ reading abilities. Each project selects and trains volunteer tutors and then deploys them 
in schools to work (usually one-on-one) with elementary school students. Most of the projects 
select and train adult tutors (federal work-study students, AmeriCorps members, and community 
volunteers), although one project is a peer-tutoring program. Other project variations include the 
length and focus of initial tutor training, the tutoring curriculum, length and frequency of tutoring 
sessions, the number of students involved in each tutoring session, the nature of on-site tutor 
supervision, the availability of in-service training opportunities for tutors, and program 
evaluation design. 

• Hillsborough Reads is a partnership between the Hillsborough Education Foundation and the 
Communities in Schools program in Tampa, Florida. The project recruits federal work-study 
students and community volunteers as tutors. Six additional AmeriCorps members are 
responsible for training tutors to implement the Building Better Readers curriculum, which 
was developed by a team of school district and local university reading specialists. All tutors 
receive two-to-three hours of initial training, and work-study students receive an additional 
six hours of training. Once on the job, tutors are supervised by an AmeriCorps member, or 
Team Leader, who works with a designated school coordinator (e.g., a lead teacher or 
assistant principal) at each school. Extended training is available to all volunteers. While the 
tutors work mostly with first-, second-, and third-graders, some specially trained Foster 
Grandparents work with kindergartners. Tutors work one-on-one with their assigned students 
twice a week for 30 minutes each session. Tutoring takes place outside of regular classroom 
reading time and is scheduled by the classroom teacher. Fluency, reading strategies, and 
other language skills are the focus of instruction. 

1998-99 Program Evaluation : Teachers completed a Student Assessment Form (pre and post) 
on all participating students to document reading achievement on a five-point scale (from 
1 unsatisfactory to 5=outstanding). Parents and teachers were surveyed at the end of the 
school year for their assessment of change in students’ reading skills and attitudes toward 
reading. In a separate experimental study, first- and second-graders experiencing difficulty 
with reading were randomly assigned to treatment (tutored) and control (non-tutored) groups. 
First-graders in both groups were tested pre and post using the Early Reading and Writing 
Assessment. Second-graders in both groups were tested pre and post on the Scholastic 
Reading Inventory. 

• Miami Reads Tutorial Project is jointly administered by the Center for Community 
Involvement at Miami-Dade Community College and by the Miami-Dade County Public 
Schools Division of Language Arts/Reading. Federal work-study students from the college 
are recruited to tutor 15-20 hours per week. Before being placed at a school, tutors attend a 
two-hour orientation and then receive four hours of training in the tutoring curriculum. This 
curriculum was developed by a team of reading specialists and classroom teachers and is 
based on the instructional models for Book Buddies and Reading Recovery. At each school, 
tutors are supervised by a reading coordinator, who selects the appropriate materials for each 
tutorial session and also is responsible for observing tutors and giving them feedback. In 
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addition, reading coordinators hold weekly conferences with tutors and first-grade teachers. 
Tutors also attend two-hour in-service training sessions every six weeks. The project focuses 
on first-graders who receive 30 minutes of instruction, two-to-three times a week, in their 
regular classrooms. In every session, tutors cover four steps: rereading familiar material, 
exploring words and sounds, writing to read, and reading new material. 

1998-99 Program Evaluation : Miami Reads received financial support from SERVE during 
its second year of operation (1998-99). Therefore, the project has evaluation data for two 
years. Early in the planning stages, project staff identified six indicators of success including 
(1) improved student achievement in reading and (2) improved student achievement in 
spelling. 5 To measure improvement in these areas, a basic battery of reading assessments was 
compiled by reading specialists in the Miami-Dade County Public Schools for use in primary 
grades tutorial programs. The assessment included tests for alphabet knowledge, concepts of 
words in print, phonemic awareness and phonics, word recognition, and reading in context. 
First-graders’ reading achievement was evaluated using pre and post measures. School-based 
personnel (reading coordinators and teachers) and reading tutors also completed surveys that 
included a few items about impact on student achievement and self-esteem. 

• Mississippi Reads is a service learning partnership of the Mississippi Institutions of Higher 
Learning in Jackson, Mississippi, and 18 college and university campuses around the state. 
Tutors include AmeriCorps members, VISTA volunteers, federal work-study students, and 
other volunteers. In most cases, tutoring sessions take place on the college campus. Campus 
supervision is usually handled by a combination of AmeriCorps members and elementary 
school teachers. Trained reading specialists conduct tutor-training sessions at every campus. 
The program targets second-, third-, and fourth-graders in Level 1 and 2 schools (schools 
rated low achieving by the state). Participating students receive two one-hour tutorial 
sessions per week. One-on-one instruction with students is done outside of their regular 
classroom reading instruction. Tutorial sessions focus on reading comprehension, sight word 
identification, and word attack skills. 

1998-99 Program Evaluation : The project gathered pretest and posttest reading achievement 
data from tutored students using the Basic Reading Inventory. Also, classroom teachers were 
surveyed regarding the impact of the project on their students’ reading comprehension, sight 
word identification, and overall academic improvement. 

• Reading Soul Mates is a peer-reading, service-learning program developed by Youth Service 
Charleston, an independent, non-profit agency in Charleston, South Carolina. Through this 
program, fourth- and fifth-graders (and occasionally sixth- to eighth-graders) are trained to 
tutor younger students, usually first- and second-graders. The older students, or Readers , 
attend three 30-minute training sessions a week for three weeks before they are matched with 
younger students, or Buddies. A combination of teachers, volunteers, federal work-study 
students, and AmeriCorps members match Readers and Buddies and then facilitate the 
tutorial sessions. Teachers are encouraged to meet with Readers at least once a week to lead 
them through reflection activities and to provide ongoing tutor training. Readers and Buddies 



5 Other indicators include stakeholder satisfaction with tutor training, tutoring materials and methods, logistical 
procedures and tutor performance, and school district assessment and selection of participants. 



meet three times a week for 30 minutes each. A typical tutorial session involves reviewing 
and summarizing material that has already been read, reading new material, answering 
comprehension questions, and writing in journals. 

1998-99 Program Evaluation : The project examined the impact of participation in the 
tutoring program on the reading ability of both Readers (tutors) and Buddies (tutees). The 
reading portion of the Metropolitan Achievement Test, 7 th edition (MAT7) was used to 
measure reading ability. First, the evaluation plan focused on grade-by-grade pretest-posttest 
gains of tutees (grades 1-3) and tutors (grades 4-8). Then a pretest-posttest control group 
design was used to assess the impact of participation on the reading ability of Readers and 
Buddies combined. The spring 1998 (prior to the beginning of the program) and spring 1999 
(end of Year 1) scores of students in the treatment and control groups were compared for 
differences in quartile shifts made by each group. The control group was composed of non- 
participants who attended the same schools as the Readers and Buddies in the treatment 
group. Surveys of classroom teachers, volunteer facilitators, and Readers provided 
additional information about the impact of the tutoring program on participating students. 

• School Reading Partners is operated by the Chapel Hill-Carrboro City Schools Office of 
Volunteer Programs in Chapel Hill, North Carolina. Volunteer tutors include students at the 
University of North Carolina and members of the community’s Retired Senior Volunteer 
Program (RSVP). Tutors attend a two-hour training session to learn more about Reading 
Recovery (on which the tutoring sessions build) and how to lead children through the 
materials and activities that the school reading specialists will select for the students. The 
school reading specialists plan the tutoring sessions and leave instructions and materials for 
the tutors to follow. District program staff members observe the tutors and provide them with 
feedback. The program targets kindergartners, first-graders that are in Reading Recovery, and 
other students who need additional help to continue improving their reading skills. In order 
for them to have more than one tutoring session a week, each student is assigned multiple 
tutors who rotate through the week. Each tutor maintains a record of what was accomplished 
in each session so that the next tutor can pick up where the last one left off. In a typical 
tutoring session, volunteers introduce and read new books and work through skill-building 
activities assigned by the reading specialist. 

1998-99 Program Evaluation : Reading assessments were administered to students for 
placement purposes only. All kindergartners in the district were given the North Carolina K-2 
Literacy Assessment. Struggling readers in the first- and second-grades were evaluated using 
Reading Recovery assessment tools. Other student impact data were collected via a telephone 
survey of classroom teachers and a focus group with site-based managers. 

As indicated above, evaluation of these five projects has varied in terms of research design and 

measurement of achievement and other impacts. The next section of this report summarizes these 

variations and related evaluation findings across projects. 



6 Thirteen schools were involved in the evaluation study. Approximately 15% of the total student population at these 
schools participated in Reading Soul Mates, which left a sizable number of students from which to select the control 
group. Criteria for being selected into the control group were not reported. 
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Program Evaluation 



The focus of all the project evaluations overall can be 
summarized by the following three evaluation questions: 



1 . 



2 . 



3 . 



Do participating students make significant gains in 
their reading skills over the course of the program? 
Do reading levels of participating students increase 
more than those of non-participating students? 

Do participating students experience other positive 
personal or academic impacts as a result of program 
participation? 



An important distinction to make 
between the evaluation studies is 
whether tutored students were 
compared with themselves before 
and after the program (as in a 
pretest-posttest design), whether 
they were compared with non- 
tutored students at the end of the 
program (a posttest-only design), 
or whether tutored and non- 
tutored students were compared 
at the beginning and end of the 
program (a pretest-posttest 
control group design). 



An important distinction to make between the evaluation 
studies is whether tutored students were compared with 
themselves before and after the program (as in a pretest- 
posttest design), whether they were compared with non- 
tutored students at the end of the program (as in a posttest-only design), or whether tutored and 
non-tutored students were compared at the beginning and the end of the program (as in a pretest- 
posttest control group design). The interpretation of results regarding the benefits of the tutoring 
project in question depends in large measure on the design that is used for the evaluation study. 
Among the examples reported here, the pretest-posttest control group designs are the strongest 
for drawing conclusions about the value of one-on-one tutoring programs. 



NOTE: Each project designed their own evaluations and produced their own reports. The 
findings that are synthesized in this report are not the product of a common cross-site evaluation 
plan. For the most part, results are presented here as they were first reported in the respective 
evaluation documents. Some findings have been re-formatted for clarity and consistency. 
Information about the reading scales and sub-scales has been included wherever possible to 
allow the reader to assess the substantive significance of the reported findings. 



Do participating students make significant gains in their reading skills over the course of the 
program? 

Four of the five projects addressed the question of whether participating students improved their 
reading skills. Of course, students are expected to improve their reading skills in the course of a 
school year. The following studies assessed how much progress tutored students made while they 
were in the program. Each study uses a different measure of improvement. 



Hillsborough Reads, Florida 

To measure gains in reading skills, the project used the Student Assessment Form, which was 
developed by project staff to document student reading achievement before and after 
participation in the Hillsborough Reads tutoring program. Classroom teachers were asked (pre 
and post) to assess students’ reading achievement on the following scale: l=Unsatisfactory, 
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2=Needs Improvement, 3=Satisfactory, 4=Good, and 5=Outstanding. This five-point scale was 
tied to benchmark standards for first and second grade. 



Forty-five percent of tutored students in grades K-3 increased their reading performance by one 
Likert-scale point on the Student Assessment Form (see Table 3). Another 17.5% progressed two 
to four points on this five-point scale. This measure took place during one school year. 



Table 3 

Hillsborough Reads 



Number of Points 
Improved on 
5-Point Scale 


Number of 
Students 
(n=407) 


Percent 
of Total 


0 


154 


38 


1 


183 


45 


2 


52 


13 


3 


16 


4 


4 


2 


.5 



At the end of the school year, a variety of stakeholders — including teachers, parents, and school 
site coordinators — were surveyed for their feedback and impressions of the tutoring program. 

• Ninety-three percent of responding teachers either agreed or strongly agreed with the 
statement: “Tutored children in my class have made progress that I believe is partially a 

o 

result of their tutoring sessions.” 

• All of the project’s 12 school site coordinators either agreed or strongly agreed that the 
Hillsborough Reads tutoring program was making a difference in students’ reading 
achievement. 

• One hundred percent of responding parents of tutored students either agreed or strongly 
agreed that they had seen progress in their child’s reading ability over the past year. 9 

Miami Reads Tutorial Project, Florida 

The project measured reading improvement using a battery of reading tests that were compiled 
by the Miami-Dade County Public Schools and recommended for use in America Reads projects 
at the primary grade level. 10 Tests include Alphabet Knowledge (recognition, sounds, 
production), Concept of Words in Print, Phonemic Awareness (sound-picture matching, spelling, 
sound-letter correspondence), Word Recognition (color and number words, pre-primer sight 
words), and Oral Reading and Comprehension. Table 4 includes the number of points needed to 
achieve the “Mastery” level on each scale or sub-scale (located in the Reading Assessments 
column). 

In both years of the project, participating first-graders showed statistically significant progress on 
all measures in the district’s basic battery of reading skills assessments. As shown in Table 4, 



7 For the Year 2 evaluation, project staff members are expanding to a seven-point scale in order to better capture 
subtle advances that occur while students are still in Level 2, the “needs improvement” stage. 

8 This result is based on a 94% response rate with 87 teachers responding out of 93 surveyed. 

9 These results are based on a 59% response rate with 147 parents responding out of 250 surveyed. 

10 The primary grades reading tests were developed by school district staff, who modeled them after assessments 
used in other elementary reading programs, including Reading Buddies and Reading Recovery. 
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participants in the Miami Reads Tutorial Project have made the greatest pre-post gains in 
phonemic awareness (spelling and letter-sound correspondence sub-scales) and word recognition 
(number words and pre-primer sight words sub-scales). In Year 2, the next cohort of first-grade 
tutees also made notable progress in phonemic awareness, word recognition, and oral reading 
and comprehension. And yet, in spite of large percentage gains in these areas, tutored students 
are still performing on average below the “mastery” level for first-graders on each of these sub- 
scales. In the case of the Oral Reading and Comprehension test, students reached only the 
“minimally adequate” level (10-14) at the end of Year 2. 



Table 4 



Miami Reads Tutorial Project 
Reading Assessment Results of First-Graders 
(District-Selected Battery of Reading Assessments) 



Reading Assessments 


1997-98 
(n=1310) 
Cohort 1 


1998-99 
(n=1020) 
Cohort 2 


Pretest 

Mean 

(std dev) 


Posttest 

Mean 

(std dev) 


Pretest 

Mean 

(std dev) 


Posttest 

Mean 

(std dev) 


Alphabet Knowledge: 

Lower case letters 

fmastery=20/261 


18.23 

( 7 . 72 ) 


24.74* 

(3.63) 


18.68 

(7.77) 


24.60* 

(3.67) 


Upper case letters 

[mastery=20/261 


18.66 

am 


25.05* 

(8.29) 


19.45 

(7.54) 


24.91* 

(6.45) 


Letters Produced 
|mastery=20/261 


17.81- 

(8.41) 


24.27* 

(4.43) 


16.85 
(Mil 


23.76* 

(5.23) 


Concept of Word in Print: 

Words in print 

[mastery-4/51 


2.83 

(1.85) 


4.64* 
a m 


3.20 

(3.25) 


4.71* 

0.79) 


Phonemic Awareness: 

Sound-picture matching 

[mastery=6/8] 


5.72 

(3.33) 


7.60* 

(1-22) 


6.22 
(4 24) 


7.62* 

(1.20) 


Spelling 

[mastery=20/261 


5.96 

(6.68) 


16.28* 

( 7 . 74 ) 


8.51 

(8.06) 


17.57* 

( 6 . 60 ) 


Sound-letter correspondence 

[mastery= 16/20] 


3.54 

(3.50) 


7.65* 

(3.14) 


7.87 

( 6 . 73 ) 


16.79* 

am 


Word Recognition: 

Color words 
[mastery=9/lll 


5.52 

(3.89) 


9.51* 

(2.83) 


5.47 

(3-82) 


9.39* 

(3.04) 


Number words 
[mastery=9/l 11 


4.43 

(4.23) 


9.60* 

(4.16) 


4.42 

(3.99) 


8.93* 

(3.28) 


Pre -primer sight words 
fmastery=29/361 


4.04 

(5.37) 


14.83* 

(6.56) 


4.49 

(6.28) 


15.65* 

( 7 . 76 ) 


Oral Reading and Comprehension: 

Benchmark book level 

[proficient= 18-201 


1.35 

(2.72) 


2.06* 

(Mil 


4.82 

(Mil 


11.56* 

am 



*A11 pretest-posttest differences in Years 1 and 2 are statistically significant at the /?= 001 level. 
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Mississippi Reads 

The project assessed reading improvement via pretest-posttest administrations of the Basic 
Reading Inventory (BRI). 11 The BRI is an informal reading assessment that includes grade-level 
word lists and reading passages. Each passage is followed by comprehension questions regarding 
topic, facts, inference, evaluation, and vocabulary. 12 Teachers also completed surveys rating 
individual student’s improvement in reading comprehension and sight word recognition. 

Table 5 shows results for 552 participating second-, third-, and fourth-graders from 32 
elementary schools in the state of Mississippi. At each of these schools, the Basic Reading 
Inventory was administered either by an AmeriCorps member or classroom teacher. At the end 
of the school year, 85% of tutored students showed gains of one or more reading grade levels. 
The other 15% showed no change. 14 



Table 5 

Mississippi Reads 

Pre/post Reading Level Increases of Tutored Students 
(Basic Reading Inventory) 



Reading 

Level 

Increases 


Percent of 
Tutored 
Students 


0 


15 


1 


24 


2 


26 


3 


19 


4 


10 


5+ 


6 



Based on the ratings of their regular classroom teachers, close to half (45-49%) of tutored 
students made “significant” to “tremendous” progress in reading comprehension and sight word 
identification (see Table 6). Forty-two percent achieved “partial” improvement in these skill 
areas, and 9-13% showed little to no improvement. 15 



Table 6 

Mississippi Reads 

Teacher Ratings of Improvement in Reading Skills 



Reading Skills 


Improvement in Reading Skills 
(Percent Responding) 


Tremendous 


Significant 


Partial 


Little or no 


Reading comprehension 


8 


37 


42 


13 


Sight word identification 


10 


39 


42 


9 



11 The Basic Reading Inventory (BRI) was developed by Dr. Jerry Johns and is published by Kendall/Hunt. 

12 Project staff members are concerned that the BRI may inflate measured reading levels. They plan to use a 
different reading assessment instrument in the future. 

13 There are no strict guidelines for administering the test. A few of the teachers who administered the test to 
Mississippi Reads participants already had some experience with the BRI. A few other teachers and all the 
AmeriCorps volunteers received some training to administer the test, but this was not required of all teachers who 
tested students for program eligibility. 

14 The Mississippi Reads model called for students to receive two hours of tutoring each week for 30-32 weeks 
(depending on the school). 

15 The number of teachers doing the rating was not available; however, ratings were received for 491 students. 
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Reading Soul Mates, South Carolina 

To assess growth in peer-tutors’ (Readers, grades 4-8) and tutees’ (Buddies, grades 1-3) reading 
ability, the project compared students’ year-to-year reading test results on the Metropolitan 
Achievement Test (MAT7). The MAT7, a nationally normed assessment, is administered by the 
Charleston County Public Schools every spring. Results on the MAT7 are expressed as national 
percentile ranks. 

Table 7 presents average MAT7 reading scores by grade for Buddies and Readers at 13 
Charleston County schools. Among the Buddies, first-graders showed by far the greatest 
improvement, jumping from the 29 th percentile nationally in 1 998 to the 45 th percentile as 
second-graders in 1999. Third-graders also improved their national percentile rankings on the 
MAT7, going from the 65 th percentile in 1998 to the 72 nd percentile as fourth-graders in 1999. 
However, the majority of Buddies, 163 second-graders, dropped from the 50 th percentile (the 
national average) to the 42 nd percentile. Percentile ranks for the fourth-grade Readers also 
dropped off between fourth and fifth grade. Readers who were in grades 5-8 in 1998 experienced 
smaller negative changes in their national percentile ranks from one grade to the next. 16 

Table 7 

Reading Soul Mates 

Change in Tutees’ and Tutors’ Average National Percentile Ranks, 1998-1999 
( Metropolitan Achievement Test, 7** Edition ) 



Grade 


No. of 
Tutees 


Average MAT7 
percentile rank 
1998 


No. of 
Tutees 


Average MAT7 
percentile rank 
1999 


Reading 

Percentile 

Gain 


1 


10 


29.0 


10 


45.0 


+16.0 


2 


163 


49.9 


163 


41.7 


-8.2 


3 


21 


65.4 


21 


71.6 


+6.2 



Summary : To determine whether participating students made progress in their reading ability, 
three of the four projects used some form of pretest-posttest skills test. The fourth project relied 
on teachers’ assessments of progress and found varying degrees of improvement in nearly two- 
thirds of the tutored students. These studies reported either modest to substantial gains in specific 
skill areas or large percentages of tutored students advancing one or more levels of the reading 
assessment used. In the peer-tutoring program, peer-tutors (grades 4-8) did not improve their 
reading skills over the course of the program. Other findings relied on the judgments of a variety 
of stakeholders, including teachers, site coordinators, and/or parents. Responding stakeholders 
generally found that participating students’ reading skills had improved, although response rates 
on stakeholder surveys were very low. In one study, teachers and site coordinators attributed 
progress in part to students’ involvement in the America Reads project. 

Do reading levels of participating students increase more than those of non-participating 
students? 



16 In the absence of standard deviations for the reported means (averages), it was not possible to assess the statistical 
significance of these findings. 
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Two of the SERVE-sponsored America Reads projects, Hillsborough Reads and Reading Soul 
Mates, used a pretest-posttest control group design to determine whether participants’ reading 
skills improved more than those of non-participants. Their results are summarized in the 
following sections. 

Hillsborough Reads, Florida 

An experimental study was conducted by an external evaluator to compare the performance of 
tutored and non-tutored first- and second-graders on several different measures, including the 
Early Reading and Writing Assessment 17 (first-graders, pre and post) and the Scholastic Reading 
Inventory 18 (second-graders, pre and post). All first-graders at School A and all second-graders at 
School B were pre-tested using the Early Reading and Writing Assessment and the Scholastic 
Reading Inventory, respectively. Students in the targeted score-range, who also met other 
academic and behavioral criteria, were randomly assigned to treatment (tutored) and control 
(non-tutored) groups. By the time of the final analyses, the treatment group contained 46 first- 
graders and 33 second-graders; the control group included 37 first-graders and 29 second- 
graders. 19 Average tutoring time was 19.87 hours for first-graders over a 20-week period and 
15.46 hours for second-graders over a 15-week period. 20 

The experimental study report cites statistically significant differences between the posttest 
means of tutored and non-tutored first-graders on two scale scores: Concepts about Print and 
Story Retelling (see Table 8). These results are based on an analysis of co-variance (ANCOVA), 
which controlled for differences in the pretest means of the tutored and non-tutored first-graders. 
No statistically significant findings were reported for the other scale scores (Writing and 
Phonemic Awareness) or for the Total Composite scores. 

Table 8 

Hillsborough Reads 

Analysis of Co-Variance of First-Graders’ Reading Scale Scores, 1998-99 
(Early Reading and Writing Assessment) 



Scales 


Tutored 

(n=46) 


Non-Tutored 
(n~37) 


ANCOVA 

Results 




Pretest 


Posttest 


Pretest 


Posttest 


F 

(P) 


Concepts about Print 


5.75 


7.30 


6.00 


6.68 


4.31 

(<•04) 


Story Retelling 


3.43 


4.46 


3.05 


3.84 


4.10 

(<.05) 



17 The Early Reading and Writing Assessment, published by Scholastic in 1996 (now out of print), is an informal test 
of concepts about print, story retelling, writing, and phonemic awareness. The Total Composite score and sub-scores 
represent the number of correct items, but the total number of items was not reported. 

18 The Scholastic Reading Inventory, available from Scholastic, is based on the Lexile Framework for Reading, 
which matches students’ reading and comprehension skills with texts of established difficulty levels. Lexile levels 
do not translate to specific graded reading levels. 

19 The student populations in School A and School B were highly mobile. For this reason, the Experimental Study 
samples at these schools experienced fairly high attrition (22% at School A and 34% at School B). 

20 There is a great deal of spread in the figures for mean hours of tutoring. Tutoring hours for the first-graders in this 
study ranged from 4 to 40 hours over 20 weeks. For second-graders, tutoring time ranged from 6 to 25 hours over 15 
weeks. The Hillsborough Reads project staff plans to focus future evaluation efforts on students who receive the 
most consistent (“substantial, sustained”) tutoring services. 
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Writing 


2.54 


3.48 


2.49 


3.57 


1.30 

(n.s.) 


Phonemic awareness 


11.11 


18.13 


11.05 


17.20 


.000 

(n.s.) 


Total Composite 21 


82.97 


100.89 


78.24 


98.38 


2.21 

(n.s.) 



NOTE: “n.s.” indicates that the F-statistic is not significant at the p<.05 level. 



The Scholastic Reading Inventory was administered pre and post to second-graders in the 
experimental study. In order to fairly assess the difference in performance of tutored and non- 
tutored students, mean posttest Lexile scores for the two groups were adjusted by pretest Lexile 
scores using ANCOVA. The ANCOVA results (see Table 9) indicate that tutored second-graders 
made significantly greater gains on the Scholastic Reading Inventory than their non-tutored 
counterparts. 



Table 9 

Hillsborough Reads 

Analysis of Co- Variance of Second-Graders’ Lexile Scores, 1998-99 
( Scholastic Reading Inventory) 





Tutored 


Non-tutored 


ANCOVA 


Score 


(n= 


33) 


(n=30) H 


Results 




Pre- 


Post- 


Pre- 


Post- 


F 




Lexile 


Lexile 


Lexile 


Lexile 


(P) 




(std dev ) 


(std dev) 


(std dev) 


(std dev) 


Lexile Reading Levels 


73.79 

(84.72) 


275.45 

(188.33) 


103.97 

(174.25) 


254.83 

(209.06) 


47.97 

(<.001) 



At the beginning of the tutoring program, all second-graders in the experimental study were 
reading at or below first-grade level on the Scholastic Reading Inventory . By the school year’s 
end, 24% of tutored students (8 out of 33) were reading on grade level (third-grade level), while 
only 7% of non-tutored students (2 out of 29) had attained this level (see Table 10). 23 

Table 10 

Hillsborough Reads 

Number of Tutored and Non-Tutored Second-Graders Reading at Each Grade Level, 1998-99 
( Scholastic Reading Inventory) 





Tutored 


Non-Tutored 


Grade Level 


(n= 


33) 


(n= 


29) 




Pre- 


Post- 


Pre- 


Post- 




Lexile 


Lexile 


Lexile 


Lexile 


Pre-primer 


27 


11 


23 


11 


First 


6 


7 


6 


7 


Second 


0 


7 


0 


9 


Third 


0 


8 


0 


2 



21 Since scale scores do not sum to the Total Composite scores, it appears that the Total Composites are based on 
performance on one or more additional scales besides those reported. 

2 The non-tutored group appears to include an outlier that inflates the group’s pre-Lexile and post-Lexile means. 
Without access to the raw data, it is not clear how this outlier affects the ANCOVA results in Table 9. 

23 Levels in the Lexile Framework used to score the Scholastic Reading Inventory do not actually correspond to 
specific grade levels; rather, they correspond to the reading comprehension levels of specific texts. It appears, 
therefore, that the project adopted its own grade-level benchmarks for Lexile levels. 
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Reading Soul Mates, South Carolina 

The performance of peer-tutors ( Readers ) and tutees ( Buddies ) on the Metropolitan Achievement 
Test (MAT7) was assessed relative to that of the total population of students at their respective 
schools. Program participants were compared to all students on the basis of the distribution of 
MAT7 scores into quartiles. Quartile distributions for 1998 and 1999 were cross-tabulated to 
look for movement into higher quartiles from the 1998 starting point. Overall, quartile 
distributions for participants and all students were very similar. An equal majority of students in 
both groups (63%) held steady in the same quartile from one year to the next. Fifteen percent of 
participants versus 17% of all students moved ahead one or more quartiles; while 23% of 
participants versus 21% of all students slipped one or more quartiles between 1998 and 1999. 24 



Table 11 

Reading Soul Mates 

Quartile Shifts in Reading Scores of Participants V. Non-Participants, 1998-1999 



Sample 


Loss 

(neg. shift 
in quartile) 


Same 
Quartile 
(1998 and 
1999) 


Gain 
(pos. shift 
in quartile) 


Participants - Readers & 
Buddies (n=584) 


23 


63 


15 


All students - participants 
and non-participants (n=4469) 


21 


63 


17 



Summary : One of the two control group studies, Hillsborough Reads, separated first- and 
second-graders in the data collection and analysis phases of the evaluation. The results indicated 
that tutored first-graders gained more than non-tutored first-graders in a few specific skill areas. 
Among second-graders, tutored students made significantly greater gains than non-tutored 
students in Lexile reading scores. The other control group study of Reading Soul Mates 
combined peer tutors (grades 4-8) and tutees (grades 1-3) in the treatment and control groups. 

The analysis was limited to a pre/post comparison of how tutors and tutees versus non-tutored 
students were distributed across reading score quartiles. No significant differences were evident 
in the two distributions. 

Do participating students experience other positive personal or academic impacts as a result of 
program participation? 

Other personal and academic impacts identified by the five projects include improved attitudes 
toward reading and school, general academic progress, improved social skills, improved self- 
esteem, increased volunteerism, and changed aspirations for the future. 

Hillsborough Reads, Florida 

At the end of the school year, parents of participating first- and second-graders were surveyed. 

All of those who responded to the survey (147 out of 250) either agreed or strongly agreed that 
they had seen positive change in their child’s attitude toward reading and/or school. 



24 Separate results comparing peer-tutors ( Readers ) with all fourth- to eighth-graders and tutees ( Buddies ) with all 
first- through third-graders were not reported. Thus, it is not possible to assess how each group fared on its own. 
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Miami Reads Tutorial Project, Florida 

For two consecutive years, school-based personnel (reading coordinators and teachers) gave the 
Miami Reads Tutorial Project high ratings for impact on students’ academic achievement and 
self-esteem. Starting with a five-point Likert scale, the distribution of ratings was translated into 
a mean for each item (see Table 12). 



Table 12 

Miami Reads Tutorial Project 

Teachers’ and Reading Coordinators’ Ratings of Program Impact on Students 



Survey Items 

[Scale: 1 ^strongly disagree to 5=strong agree] 


1997-98 

(n=143) 


1998-99 

(n=166) 


Mean 

(std dev) 


Mean 

(std dev) 


The America Reads program had a strong impact 


4.01 


4.23 


on the academic achievement of my students. 


( 1 . 19 ) 


(. 90 ) 


The America Reads program had a strong impact 


4.23 


4.43 


on the self-esteem of my students. 


(111) 


(. 90 ) 



Reading tutors have also given the project high ratings for impact on students’ academic 
achievement and self-esteem (see Table 13). 



Table 13 

Miami Reads Tutorial Project 
Tutors’ Ratings of Program Impact on Students 



Survey Items 

[Scale: l=strongly disagree to S=strong agree] 


1997-98 

(n=143) 


1998-99 

(n=166) 


Mean 

(std dev) 


Mean 

(std dev) 


The America Reads program had a strong impact on 


4.68 


4.74 


the academic achievement of the children I tutored. 


(. 50 ) 


(. 47 ) 


The America Reads program had a strong impact on 


4.71 


4.78 


the self-esteem of the children I tutored. 


(. 50 ) 


(. 45 ) 



Mississippi Reads 

Teachers were asked to rate each individual student’s overall academic improvement over the 
course of the tutoring sessions. 25 According to their regular classroom teachers, most students 
made at least partial improvement in their overall academic performance. Teachers reported that 
43% of tutored students had shown “significant” or “tremendous” improvement. Partial 
improvement was reported for 46% of tutored students. Eleven percent of participants showed 
little or no improvement in their overall academic performance. 

Reading Soul Mates, South Carolina 

Teachers, adult volunteers, and Readers (tutors) were surveyed to determine the impact of the 
RSM program on its academic, social, and civic goals. Response rates were low. 26 What follows 
is a brief summary of the kinds of observations made by each group: 



25 The number of teachers doing the rating was not available; however, ratings were received for 491 students. 

26 Out of about 26 Buddy teachers (grades 1-3), three responded; these three teachers were responsible for 48 (out of 
3 1 0) Buddies. Six out of approximately 26 Reader teachers (grades 4-8) responded; these six teachers were 
responsible for 206 (out of 318) Readers. Three out of 25 AmeriCorps members responded to program surveys. And 
out of 318 Readers, 107 responded to the pre-survey, and 88 responded to the post-survey. 
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• About the Buddies: Two of three teachers (grades 1-3) who returned surveys saw 
improvement in the Buddies ’ social skills, such as learning to be patient, listening better, 
communicating more, and taking pride in their work. 

• About the Readers : Five of six responding teachers (grades 4-8) acknowledged improvement 
in social skills, especially that the Readers’ sense of responsibility, maturity and 
communication skills had improved. In a shorter follow-up survey, other teachers of Readers 
(number unknown) estimated that 90-98% of their students had achieved the academic, 
social, and civic goals of the RSM program. Three AmeriCorps members reported that the 
Readers ’ social skills had improved. 

• According to the Readers themselves: There was little change from pre to post in terms of the 
highest level of school the students plan to complete; the modal response (81-85%) was 
“graduate from college/technical training.” The percentage of students interested in teaching 
as a career doubled from 9 to 18%. There was a 10% increase in the number of Readers who 
recognized their own leadership capacity. Finally, the number of Readers who reported 
volunteering every day increased from 1 7 to 22%, while the number who said they never 
volunteered decreased from 36 to 3 1%. 

School Reading Partners, North Carolina 

A group of classroom teachers, who were interviewed by telephone, all reported enthusiastically 
that their students had benefited from participation in School Reading Partners. Most of them 
identified the social aspect of the one-on-one sessions — rather than any academic gains — as 
having been most salient for their students. One teacher, however, did express the belief that the 
tutoring sessions helped students by providing a good review of the materials that also were used 
in the Reading Recovery program. 

In a focus group setting, site-based managers noted both social and academic benefits for 
students of one-on-one attention from tutors. According to the site-based managers, the children 
“feel special” and “gain the opportunity to form a trusting relationship with an adult.” They also 
noted that volunteers serve as positive role models for the children they tutor. Among the 
academic benefits cited in the focus group were improvement in reading skill levels, spending 
more time-on-task, having more opportunities for language development as a result of interacting 
with an adult, and having opportunities to generalize the skills they have learned in the 
classroom. In particular, the site-based managers noted that kindergartners, ESL, and Reading 
Recovery students appeared to benefit the most from the tutoring program. 

Summary : The studies summarized above relied on the judgments of school personnel, parents, 
adult tutors, and peer-tutors to assess a variety of other impacts on students besides improved 
reading skills. In most cases, the response rates were low, so the results should be viewed 
cautiously. Suffice it to say that the results of these studies suggest a number of other possible 
impacts (e.g., general academic progress, improved self-esteem) that might be studied more 
rigorously in the future. In the case of peer-tutoring programs, the peer-tutors’ social skills and 
educational aspirations might also be affected by participation in these programs. 
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Program Evaluation Conclusions 



From the evaluations of the five America Reads .■ 

projects, we have learned the following: Largely through surveys of I 

parents, teachers, and sometimes 

Do participating students make significant gains in tutors, each of the five projects 

their reading skills over the course of the program? was . abl ® t0 re ^?^ t that * b,dents 

achieved something positive (other 

T . than higher test scores) during the 

In most cases, tutored students continued to make sch00l year whlch they „ ere 

progress in reading while they were in the tutoring tutored. Their accomplishments 

programs. Of the four projects that looked at pretest- included improved attitudes 

posttest differences on some form of quantitative toward reading and/or school, 

measure (staff-developed scale, standard battery of overall academic improvement, 

reading assessments, informal reading inventory, or higher self-esteem, and better 

standardized reading test), three of them indicated that social skil,s - 

tutored students’ reading skills increased while they 

were in the program. In essence, this means that students were not held back by the projects. It 
does not mean that the projects were responsible for gains in achievement or that there were 
additional gains made by the tutees. 

Do reading levels of participating students increase more than those of non-participating 
students? 

Two of the five projects used a pretest-posttest control group design to compare the reading 
achievement of tutored and non-tutored students. Of the two studies, only one provides evidence 
in support of the positive impact of volunteer tutoring programs. The other study concludes that 
there are no differences between participants and non-participants. 

Do participating students experience other positive personal or academic impacts as a result of 
program participation? 

Largely through surveys of parents, teachers, and sometimes tutors, each of the five projects was 
able to report that students achieved something positive (other than higher test scores) during the 
school year in which they were tutored. Their accomplishments included improved attitudes 
toward reading and/or school, overall academic improvement, higher self-esteem, and better 
social skills. In the one peer-tutoring project, at the end of the school year, a greater percentage 
of peer-tutors had expressed interest in teaching as a career, had volunteered in their 
communities, and saw themselves as leaders compared to where they stood on these issues at the 
beginning of the year. Sometimes survey questions in these studies were phrased to link positive 
attitudes and behaviors to participation in the tutoring programs, but not always. In any case, 
results from three of the five studies were based on low response rates and/or sample sizes and 
are, therefore, suggestive at best. 



Largely through surveys of 
parents, teachers, and sometimes 
tutors, each of the five projects 
was able to report that students 
achieved something positive (other 
than higher test scores) during the 
school year in which they were 
tutored. Their accomplishments 
included improved attitudes 
toward reading and/or school, 
overall academic improvement, 
higher self-esteem, and better 
social skills. 
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Program Evaluation Recommendations 



Evaluation is often as much an art as it is a science. One of the benefits of synthesizing 
individual project evaluations is to identify fresh and “artful” ideas for addressing common 
questions across projects. The following suggestions include some of the best evaluation 
practices currently in use by the SERVE-sponsored America Reads projects. 

Future evaluations of America Reads projects should include design comparisons that enable 
projects to make clear statements about their value for helping students learn to read. Two of the 
studies reviewed here used pretest-posttest control groups. Both of these studies had the “luxury” 
of assigning students to the tutored and non-tutored groups because there were more eligible 
students than tutors in the program. Logistical and cost factors often prohibit this kind of 
evaluation. When it isn’t possible to randomly select a control group, there are other kinds of 
comparisons that can be made, including the following suggestions: 

• Matched Comparison Groups - Find a similar school that isn’t already participating in your 
project. Urban/rural/suburban location and student demographics are two common factors 
used to identify similar schools. Ask the principal to allow you to collect pretest and posttest 
data from a group of students who meet the criteria for participation in your program. After 
selecting students based on program eligibility, match participants and non-participants on 
specific relevant criteria, such as standardized test scores and free lunch status. This kind of 
matched comparison group is often the next best thing to a randomized control group. 

• Dosage Effects - Depending on the reliability of tutors and other program implementation 
factors, students don’t always receive the same amount of tutoring. While regrettable, this 
situation actually provides the opportunity to make another kind of comparison — between 
students who receive a low level of tutoring (e.g., less than 10 hours during the school year) 
and those who receive a high level (e.g., more than 50 hours a year). For example, you could 
calculate pretest-posttest gains for the two groups (one with a low number of tutoring hours 
and the other with a high number of tutoring hours) and compare the means and standard 
deviations of the two groups. A more sophisticated analyst might make use of regression or 
analysis of co-variance (ANCOVA) techniques to calculate the effect of an increasing 
amount of tutoring (e.g., 1-60+ hours) on student achievement while controlling or adjusting 
for pretest performance. In this case, a positive result would indicate that the more tutoring a 
student has, the better he or she scores on the reading assessment. 

• Performance Targets - Another kind of comparison is between participants’ performance 
and a set of well-crafted, validated benchmarks. Experienced educators should be able to 
review the past performance of different cohorts of students (who did not receive tutoring 
services) and determine a reasonable and challenging level of gain for students who do 
receive tutoring services. The point is to be able to say that non-tutored students (who match 
your program’s eligibility criteria) typically perform at Level A by year’s end, but tutored 
students performed at Levels B or C or D at the end of the school year. 

• Stakeholder Attribution - In the absence of other more “objective” forms of comparison, you 
should at least ask parents, teachers, and other stakeholders to reflect on the degree to which 
(in their opinions) program participation influenced reading improvement. This usually 
involves a two-stage question. First, you ask stakeholders and other observers whether they 
saw any improvement on some specific dimension of reading that your program is designed 



to address (e.g., fluency and expression, 
comprehension, spelling, self-confidence about 
reading, etc.). Then you ask them to indicate whether 
improvement was related to project participation (e.g., 
yes, no, I don’t know) or to what degree was the 
project responsible for the improvement they observed 
(e.g., not at all, somewhat, a lot). If possible, have 
these observers answer the question for individual 
students, as opposed to participating students in 
general. 

Other recommendations include (1) selecting a few key 
questions and finding multiple (two or more) sources of 
data to address each one, (2) encouraging a high 
proportion of respondents to return surveys, and (3) 
reporting on the substantive, as well as the statistical, 
significance of results. 



Another strategy that works 
well with some groups is to 
provide tangible incentives for 
responding. If you can’t afford 
to send something to every 
respondent, you can award 
“prizes” to the first few people 
who respond. For teachers, 
you might award a set of 
instructional materials for 
their classrooms. Parents 
might respond well to a gift 
certificate at a local children’s 
bookstore. Be creative and, 
whenever possible, seek 
donated prizes. 



• Triangulating Evidence - The most convincing evaluation findings are those based on 
multiple sources of evidence. “Triangulation” literally means bringing three or more pieces 
of evidence to bear on a question. In small projects, this level of attention to any single 
indicator is not realistic. However, it is important to seek at least one other source of support 
for your findings. Some measures can stand on their own pretty effectively — such as a valid 
and reliable test of reading achievement. But softer measures, like participant attitude surveys 
or self-reports of reading improvement, should be supplemented with other perspectives. For 
example, teachers and parents can also be asked to report on changes they have observed in 
students’ reading attitudes and behaviors. The point is to not rely on one single source of 
evidence when you are making statements about specific aspects of project impact. 

• Raising Survey Response Rates - With all of the requests for information that the average 
citizen gets nowadays, it is important to set aside time and other resources to make sure that 
your target audience responds to your survey. There are a number of strategies for doing this. 
For example, automatically send everyone a “thank you’Vreminder postcard shortly before 
the survey due date. Randomly select a small sample (10-20%) of potential respondents, call 
them on the phone, and go through the survey with them. Another strategy that works well 
with some groups is to provide tangible incentives for responding. If you can’t afford to send 
something to every respondent, you can award “prizes” to the first few people who respond. 
For teachers, you might award a set of instructional materials for their classrooms. Parents 
might respond well to a gift certificate at a local children’s bookstore. Be creative and, 
whenever possible, seek donated prizes. 

• Reporting Substantive Significance - Oftentimes, studies report statistical significance but 
do not explain what the numbers really mean. The fact is that a statistically significant result 
may not indicate meaningful change. Take for example the following scale for grade-level 
reading improvement: l-4=pre-primer, 5-8=first grade, 9-12=second grade, and 13-16=third 
grade. The goal of your program is to help students read independently and well at each 
grade level. However, you recognize that some students start off at too low a level to 
realistically achieve this in a year. So you stipulate that, wherever they start, your participants 



will achieve at least one grade-level increase while they 
are in the program. In analyzing results for a group of 
second-graders, you find that the group mean is 5 on the 
pretest at the beginning of second grade and 7 on the 
posttest at the end of the school year. Given a large and 
stable-enough sample, this two-point increase may be 
statistically significant. After all, a statistically significant 
difference really only means that the difference between 



test scores is not zero. 27 But the bottom line is that your 



Some measures can stand 
on their own pretty 
effectively — such as a 
valid and reliable test of 
reading achievement. But 
softer measures, like 
participant attitude 
surveys or self-reports of 
reading improvement, 
should be supplemented 
with other perspectives. 



students have improved their reading by only half of a 
grade level and are still reading on the first-grade level to 
boot. In such a case, this two-point increase is not 
substantively significant. On the other hand, you may also get results that appear to be very 
meaningful in terms of the scale you are using, but are not statistically significant due to a 
small sample and wide variation in scores. For this reason, it is important to consider both 
substantive and statistical significance when interpreting your results. 



27 Put another way, the absolute value of the difference between group means (pretest v. posttest) is greater than 
zero. 
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Program Implementation Recommendations 



Past research indicates that volunteer tutoring programs can 
have a positive impact on students’ reading ability: 

• In a study of the Howard Street Tutoring Program in 
Chicago, participating second- and third-graders 
achieved greater gains than non-participants on all 
measures in a battery of word recognition, spelling, and 
reading tests. Participants were tutored an average of 50 
hours each by adult volunteers. 

• After a year of tutoring by adult volunteers, a sample of 
participants in the now-defunct Dade County School 
Volunteer Development Project (Florida) made greater gains than non-participants on the 
Metropolitan Achievement Test. 

Both of these studies used a pretest-posttest control group design, with students being randomly 
assigned to the treatment and control groups. 28 Peer-tutoring programs have achieved similar 
results. In a meta-analysis of findings for peer-tutoring programs in reading and mathematics, 
peer-tutors and tutees scored higher than non-participating control groups in 52 out of 65 studies 
on quantitative measures of achievement. 29 

The America Reads projects in the SERVE region have been designed to incorporate many of 
the components of promising and proven programs, such as training and feedback for tutors, 
structured lesson plans for tutors to follow, and frequent (2-3 times a week) tutoring sessions. 30 If 
previous evaluations demonstrate the effectiveness of volunteer tutoring programs, and if the 
SERVE programs include components that are known to be effective, the next step is to ensure 
that these components are implemented well. The America Reads projects at SERVE have 
already devoted careful attention to developing and revising their training components. For some 
projects, the next step is to refine program implementation. 

In focus group discussions with SERVE staff in November 1998, America Reads project 
coordinators from the nine funded sites identified a number of “challenges” which affect the 
implementation of their tutoring components. Some of these challenges have also been identified 
by the evaluators who studied the projects. 



In a meta-analysis of 
findings for peer-tutoring 
programs in reading and 
mathematics, peer-tutors 
and tutees scored higher 
than non-participating 
control groups in 52 out of 
65 studies on quantitative 
measures of achievement 
(Cohen et al., 1982). 



28 Wasik, Barbara A. (1997). Volunteer Tutoring Programs: A Review of Research on Achievement Outcomes 
(Report No. 14). Baltimore, MD: Center for Research on the Education of Students Placed at Risk [available at 
http://www.csos.ihu.edU/crespar/Reports/ l. 

29 The results for mathematics tutor programs were slightly stronger than those for reading tutor programs. See 
Cohen, Peter A., Kulik, James A., and Kulik, Chen-Li S. (1982). “Educational Outcomes of Tutoring: A Meta- 
analysis of Findings.” American Educational Research Journal, 19 (2), pp.237-248. 

30 For more on the components of effective programs, see Potter, Jana; Blankenship, Judy; and Carlsmith, Laura 
(1999). So That Every Child Can Read: A Review of Effective and Promising Practices in Volunteer Reading 
Tutoring Programs. Portland, OR: Northwest Regional Educational Laboratory. See Also Wasik, Barbara A. (1998). 
“Using Volunteers as Reading Tutors: Guidelines for Successful Practices.” The Reading Teacher, 51 (7), pp. 562- 
570. 
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Tutoring project 
coordinators must clearly 
explain to school staff: 

“We’re supplementing and 
reinforcing what you’re 
doing. We’re not trying to 
take the place of a certified 
professional. We’re just 
doing our part.” 



Tutor Transportation - Work-study students typically 
don’t have their own transportation; in fact, they may 
even be prohibited from having a car on campus if they 
receive federal work-study funds. Some schools are 
located close to college campuses, but the schools 
usually in greatest need of assistance are usually more 
isolated. In such cases, even public transportation is not 
available to get tutors to their assigned schools on time. 

Volunteer Work Ethic - The tutoring job is the first paid 
work for some work-study students who don’t yet have 
the maturity or sense of responsibility to show up on 
time and conduct themselves in a professional way. Projects want to set standards like 
“Show up or lose the job,” but they don’t feel they can afford to do this once the 
tutors have been trained. So they feel like they have to relax their standards and be 
happy with whatever time the tutors do put in. 

School Expectations - Schools expect too much of the volunteers, so project 
coordinators have to be clear about what the schools can expect. One coordinator tries 
to be as clear as possible during initial meetings with school staff: “We have to go 
back and say to [school people, ‘The volunteers are] learning to meet [students’] 
needs, but they cannot fulfill the primary needs of your school system.’ And I always 
have to say, ‘We are supplementing and reinforcing what you’re doing. We’re not 
trying to take the place of a certified professional. We’re just doing our part. This is 
what we bring to the table.’” 

Resistance to Change - It is very difficult to be a change agent in communities that 
are not used to change. Especially if you are not a native of that community, you have 
to be very patient. One coordinator said, “We have to be sure that we back off and 
give them an opportunity to think about their own needs, especially in rural areas. ... 
So, the best thing to do is just to wait and let them conceptualize it, and meet and 
greet and answer questions, and be very patient with them until they’re able to 
understand that you’re not somebody who’s coming in to belittle them because their 
students cannot read as well as everybody else in the whole wide world.” 

Quality Control and Participant Autonomy - Some coordinators find it difficult to 
ensure that schools adhere to a specific model when projects are geographically 
spread out: “It’s finding a balance between micro-managing or identifying and 
implementing a model and allowing for creativity and freedom. ... We have to 
identify a framework, then allow people to build their model from that framework.” 
Participant Buy-In - Initial buy-in is, of course, important at the beginning of the 
program. But project coordinators also need to pay some attention to bringing along 
new district administrators, principals, and teachers as they enter the picture. In the 
case of a new district-level reading specialist, one coordinator noted, “We can’t move 
forward much without her leadership,” but this is a challenge when new personnel are 
also trying to learn about all the other programs for which they are responsible. 



Addressing these challenges and other program implementation issues will require a mix of 
technical assistance, creativity, and funding — and project coordinators are looking to SERVE for 
support. 



Program Implementation Support 



Project coordinators reported being pleased to have 
opportunities through SERVE to network with each other 
and share ideas. In addition, they requested the following 
kinds of support: 

• Commendation letters from SERVE to high-level 
administrators at the projects’ host institutions, “...I 
think sometimes with our program we’re better known 
around the country than we are within our own 
institution.” 

• Press releases from SERVE to local media that put the 
local America Reads effort into the regional and 
national contexts for improving literacy: “...what it 
means to the citizens of this country and how [we’re] a 
part of it.” 

• Send SERVE staff to the projects for site visits that 
generate visibility for the projects. 

• Publicize what the nine grant-funded projects are doing in high-quality reports so that 
“other people who didn’t get the grant at least get the information out of this.” 

• Create a document modeled on Preventing Reading Difficulties in Young Children 
that addresses “Best Practices” supported by evaluation results among programs that 
are using volunteers for literacy instruction. 

• Present findings about the SERVE-funded projects at regional and national meetings 
concerning volunteerism and service learning: ’’SERVE can apply to present it, and 
we can all come and help.” 

• Continue to hold meetings that bring together a diverse group of people, from 
“community-based organizations, to state, to university-oriented, to school-site 
oriented.” 

• Create a special travel fund that enables project coordinators to visit each other and 
see first-hand what’s going on in other projects. 

• Support a third-party evaluation for the purpose of helping coordinators improve their 
projects and generate information that will help them get additional funding. 

To the Lab’s credit, SERVE has already responded to a number of these requests since 
the project coordinator focus groups were conducted. Project support has included 
making presentations about the America Reads Challenge grant projects at regional and 
national meetings, continuing to host networking and information sharing meetings for 
diverse project partners, and supporting this introductory guide to implementing and 
evaluating America Reads tutoring programs. 



SERVE has already responded 
to a number of these requests 
since the project coordinator 
focus groups were conducted. 
Project support has included 
making presentations about the 
America Reads Challenge grant 
projects at regional and national 
meetings, continuing to host 
networking and information- 
sharing meetings for diverse 
project partners, and 
supporting this introductory 
guide to implementing and 
evaluating America Reads 
tutoring programs. 
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Project Contact Information 

Hillsborough Reads 

Hillsborough Education Foundation 

2010 East Hillsborough Avenue, Suite 212 

Tampa, FL 33610 

Phone: (813) 231-1904 

Fax: (813)231-1905 

Miami Reads Tutorial Project 
Miami-Dade Community College 
Center for Community Involvement 
300 NE 2 nd Avenue, Room 3116 
Miami, FL 33132 
Phone: (305) 237-7477 
Fax: (305) 237-7580 

Mississippi Reads 

Mississippi Institutions of Higher Learning 
3825 Ridgewood Road, Suite 610 
Jackson, MS 39211-6489 
Phone: (601) 432-6291 
Fax: (601)432-6982 

Reading Soul Mates 
Youth Service Charleston, Inc. 

P.O. Box 22085 
Charleston, SC 29401 
Phone: (843) 937-6517 
Fax: (843) 937-6524 

School Reading Partners 

Chapel Hill-Carrboro City Schools 

Lincoln Center 

750 Merritt Mill Road 

Chapel Hill, NC 27516 

Phone: (919) 967-8211, x 281 

Fax: (919) 933-4560 
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